Large internet scale services may have significant topology complexity on the service endpoint side and may have significant service manager fan-out on the service management side. For example, on the service endpoint side (e.g., front end) a large internet scale service may provide its service to tens of millions of consumers using hundreds or thousands of servers. The servers may be organized into pools of servers. Pools of servers may be members of virtual servers. Consumers may not have direct access to members of the pools of servers but rather may have access to a virtual server(s). The virtual server(s) may receive requests from consumers and then route the requests to servers in the pools of servers. A load balancer may be a device or process that routes traffic through a virtual server to an appropriate pool and ultimately to a member server. The front-end topology may become even more complicated when servers appear in more than one pool, when load balancers distribute requests to overlapping sets of servers, when load balancers are arranged in a multi-layer topology, when servers are arranged in a multi-layer topology, when servers provided by different vendors have different capabilities, when load balancers provided by different vendors have different capabilities, when servers or load balancers are added or removed, or for other reasons.
Conventionally, managing server membership in a server pool(s) has been a challenge. Similarly, managing an individual server, virtual server, server pool, or load balancer may have been a challenge. More generally, managing service endpoint devices has been a challenge. The challenges may have been exacerbated by the fan-out on the server management side. For example, to provide 24/7/365 service, a large internet scale service provider may have multiple teams of human service managers and multiple instances of automated service management applications attending the service endpoint devices (e.g., servers, load balancers). Not all service managers may be instantly aware of decisions made by all other service managers. Additionally, not all service managers may have the same information. For example, a first service manager in a first location may have a first set of information about a server or pool of servers while a second service manager in a second location may have a second different set of information about the server or pool of servers. The first set of information may not agree with the second set of information. Thus, the two service managers may simultaneously make different decisions about how to manage a server in a pool of servers.